A neural blackboard architecture of sentence structure

We present a neural architecture for sentence representation. Sentences are represented in terms of word representations as constituents. A word representation consists of a neural assembly distributed over the brain. Sentence representation does not result from associations between neural word assemblies. Instead, word assemblies are embedded in a neural architecture, in which the structural (thematic) relations between words can be represented. Arbitrary thematic relations between arguments and verbs can be represented. Arguments can consist of nouns and phrases, as in sentences with relative clauses. A number of sentences can be stored simultaneously in this architecture. We simulate how probe questions about thematic relations can be answered. We discuss how differences in sentence complexity, such as the difference between subject-extracted versus object-extracted relative clauses and the difference between right-branching versus center-embedded structures, can be related to the underlying neural dynamics of the model. Finally, we illustrate how memory capacity for sentence representation can be related to the nature of reverberating neural activity, which is used to store information temporarily in this architecture

Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking

Deep neural networks have become a primary tool for solving problems in many fields. They are also used for addressing information retrieval problems and show strong performance in several tasks. Training these models requires large, representative datasets and for most IR tasks, such data contains sensitive information from users. Privacy and confidentiality concerns prevent many data owners from sharing the data, thus today the research community can only benefit from research on large-scale datasets in a limited manner. In this paper, we discuss privacy preserving mimic learning, i.e., using predictions from a privacy preserving trained model instead of labels from the original sensitive training data as a supervision signal. We present the results of preliminary experiments in which we apply the idea of mimic learning and privacy preserving mimic learning for the task of document re-ranking as one of the core IR tasks. This research is a step toward laying the ground for enabling researchers from data-rich environments to share knowledge learned from actual users' data, which should facilitate research collaborations.Comment: SIGIR 2017 Workshop on Neural Information Retrieval (Neu-IR'17)}{}{August 7--11, 2017, Shinjuku, Tokyo, Japa

pMIIND-an MPI-based population density simulation framework

MIIND [1] is the first publicly available implementation of population density algorithms. Like neural mass models, they model at the population level, rather than that of individual neurons, but unlike neural mass models, they consider the full neuronal state space. The central concept is a population density, a probability distribution function that represents the probability of a neuron being in a certain part of state space. Neurons will move through state space by their own intrinsic dynamics or driven by synaptic input. When individual spikes do not matter but only population averaged quantities are considered, these methods outperform direct simulations using neuron point models by a factor 10 or more, whilst (at the population level) producing identical results to simulations of spiking neurons. This is in general not true for neural mass models. Population density methods also relate closely to analytic evaluations of population dynamics. The evolution of the density function is given by a partial differential equation (PDE). In [3] a generic method was presented for solving this equation efficiently, both for small synaptic efficacies (diffusion limit; the PDE becomes a Fokker-Planck equation) and for large ones (finite jumps). We demonstrated that for leaky-integrate-and-fire (LIF) neurons this method reproduces analytic results [1] and uses of the order of 0.2 s to model 1s simulation time of infinitely large population of spiking LIF neurons. We now have developed this method to apply to any 1D neuron point model [3], not just LIF neurons and demonstrated the technique on quadratic-integrate-and-fire neurons. We are therefore in the position to model large heterogeneous networks of spiking neurons very efficiently. A potential bottleneck is MIIND's serial simulation loop. We developed an MPI implementation of MIIND's central simulation loop starting from a fresh code base, and addressed serialization, which is now done at the level of individual cores. Central assumption in the set up is that firing rates are communicated, not individual spikes, so bandwidth requirements are low. Latency is potentially a problem, but with the use of latency hiding techniques good scalability for up to 64 cores has been achieved ondedicated clusters. The scalability was verified with a simple model of cortical waves in a hexagonal network of populations with balanced excitation-inhibition. pMIIND is available on Sourceforge, through its git repository: git://http://miind.sourceforge.net A CMake-based install procedure is provided. Since pMIIND is set up as a C++ framework, it is possible to define one's own algorithms and still take advantage of the MPI-based simulation loop

The state of MIIND

MIIND (Multiple Interacting Instantiations of Neural Dynamics) is a highly modular multi-level C++ framework, that aims to shorten the development time for models in Cognitive Neuroscience (CNS). It offers reusable code modules (libraries of classes and functions) aimed at solving problems that occur repeatedly in modelling, but tries not to impose a specific modelling philosophy or methodology. At the lowest level, it offers support for the implementation of sparse networks. For example, the library SparseImplementationLib supports sparse random networks and the library LayerMappingLib can be used for sparse regular networks of filter-like operators. The library DynamicLib, which builds on top of the library SparseImplementationLib, offers a generic framework for simulating network processes. Presently, several specific network process implementations are provided in MIIND: the Wilson–Cowan and Ornstein–Uhlenbeck type, and population density techniques for leaky-integrate-and-fire neurons driven by Poisson input. A design principle of MIIND is to support detailing: the refinement of an originally simple model into a form where more biological detail is included. Another design principle is extensibility: the reuse of an existing model in a larger, more extended one. One of the main uses of MIIND so far has been the instantiation of neural models of visual attention. Recently, we have added a library for implementing biologically-inspired models of artificial vision, such as HMAX and recent successors. In the long run we hope to be able to apply suitably adapted neuronal mechanisms of attention to these artificial models

Combinatorial structures and processing in Neural Blackboard Architectures

We discuss and illustrate Neural Blackboard Architectures (NBAs) as the basis for variable binding and combinatorial processing the brain. We focus on the NBA for sentence structure. NBAs are based on the notion that conceptual representations are in situ, hence cannot be copied or transported. Novel combinatorial struc- tures can be formed with these representations by embedding them in NBAs. We discuss and illustrate the main characteristics of this form of combinatorial pro- cessing. We also illustrate the NBA for sentence structures by simulating neural activity as found in recently reported intracranial brain observations. Furthermore, we will show how the NBA can account for ambiguity resolution and garden path effects in sentence processing

The role of recurrent networks in neural architectures of grounded cognition: learning of control

Recurrent networks have been used as neural models of language processing, with mixed results. Here, we discuss the role of recurrent networks in a neural architecture of grounded cognition. In particular, we discuss how the control of binding in this architecture can be learned. We trained a simple recurrent network (SRN) and a feedforward network (FFN) for this task. The results show that information from the architecture is needed as input for these networks to learn control of binding. Thus, both control systems are recurrent. We found that the recurrent system consisting of the architecture and an SRN or an FFN as a "core" can learn basic (but recursive) sentence structures. Problems with control of binding arise when the system with the SRN is tested on number of new sentence structures. In contrast, control of binding for these structures succeeds with the FFN. Yet, for some structures with (unlimited) embeddings, difficulties arise due to dynamical binding conflicts in the architecture itself. In closing, we discuss potential future developments of the architecture presented here

GATology for Linguistics: What Syntactic Dependencies It Knows

Graph Attention Network (GAT) is a graph neural network which is one of the strategies for modeling and representing explicit syntactic knowledge and can work with pre-trained models, such as BERT, in downstream tasks. Currently, there is still a lack of investigation into how GAT learns syntactic knowledge from the perspective of model structure. As one of the strategies for modeling explicit syntactic knowledge, GAT and BERT have never been applied and discussed in Machine Translation (MT) scenarios. We design a dependency relation prediction task to study how GAT learns syntactic knowledge of three languages as a function of the number of attention heads and layers. We also use a paired t-test and F1-score to clarify the differences in syntactic dependency prediction between GAT and BERT fine-tuned by the MT task (MT-B). The experiments show that better performance can be achieved by appropriately increasing the number of attention heads with two GAT layers. With more than two layers, learning suffers. Moreover, GAT is more competitive in training speed and syntactic dependency prediction than MT-B, which may reveal a better incorporation of modeling explicit syntactic knowledge and the possibility of combining GAT and BERT in the MT tasks